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Abstract 

Reconfiguration,  on-line  maintenance,  load  balancing,  and  similar  operations  on  complex  software 
require  (among  other  things)  dynamic  binding  of  communicating  peers,  so  that  modules  can  be  installed 
and  removed  in  a  framework  of  other  functional  modules.  A  binding  may  take  the  form  of  TCP/IP 
protocol  connection,  remote  procedure  call  server  location,  shared  memory  address  resolution,  or  object 
handle  resolution,  depending  on  the  underlying  communication  mechanism.  However.  the  need  for 
abstraction  and  modularity  is  independent  of  the  particular  IPC  mechanism(s)  being  used.  A  module 
should  not  need  to  know  how  it  is  used,  specifically,  how  it  is  bound  together  with  other  modules  to 
implement  a  more  complex  function. 

Therefore,  the  bindings  between  two  modules  A  and  B  should  in  general  be  created  and  destroyed  by 
a  third  module  C .  We  call  this  facility  a  third  party  connect.  The  module  C  should  not  have  to  have  any 
particular  relationship  (other  than  authentication)  to  modules  A  and  B  in  order  to  dynamically  connect  and 
disconnect  them.  In  particular,  it  should  not  have  to  be  one  of  A  and  B.  be  a  common  ancestor  of  them,  or 
have  an  existing  binding  to  them.  Similarly,  neither  A  nor  B  should  have  to  take  an  active  part  in 
establishing  or  tearing  down  bindings. 

Most  IPC  mechanisms  either  do  not  support  third  party  connects  at  all  (e.g.,  TCP/IP.  Accent),  do  not 
allow  dynamic  reconfiguration  (e.g..  UNIX),  or  give  binding  authority  to  only  a  fixed  internal  element  of 
the  system  (e.g.,  remote  procedure  call).  Designers  of  future  network  communication  protocols  and 
interprocess  communication  mechanisms  should  provide  for  more  flexible  session  layer  features  in  general, 
third  party  connect  in  particular. 


This  work  was  supported  in  part  by  U.S.  Army  Engineering  Topographic  Laboratories  research  grant 
no.  DACA76-85-C-001.  We  thank  the  Xerox  Corporation  University  Grants  Program  for  providing 
equipment  used  in  the  preparation  of  this  paper. 
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20.  ABSTRACT  (Continued) 


--not  need  to  know  how  it  is  used  (specifically,  how  it  is  bound  together  with 
other  modules  to  implement  a  more  complex  function). 

Therefore,  the  bindings  between  two  modules  A  and  B  should  in  general  be  created 
and  destroyed  by  a  third  module  C.  We  call  this  facility  a  third  party  connect. 
The  module  C  should  not  have  to  have  any  particular  relationship  (other  than 
authentication)  to  modules  A  and  B  in  order  to  dynamically  connect  and  disconnect 
them.  In  particular,  it  should  not  have  to  be  one  of  A  and  B,  be  a  common 
ancestor  of  them,  or  have  an  existing  binding  to  them.  Similarly,  neither  A  nor 
B  should  have  to  take  an  active  part  in  establishing  or  tearing  down  bindings. 

x. 

Most  I  PC  mechanisms  either  do  not  support  third  party  connects  at  all  (e.q., 
TCP/IP,  Accent),  do  not  allow  dynamic  reconfiguration  (e.q.,  UNIX),  or  give 
binding  authority  to  only  a  fixed  internal  element  of  the  system  (e.g.,  remote 
procedure  call).  Designers  of  future  network  communication  protocols  and 
interprocess  communication  mechanic. . is  should  provide  for  more  flexible  session 
layer  features  in  general,  third  party  connect  in  particular. 


1.  Introduction 

Many  in\ estigations  of  distributed  programming  start  with  a  particular  interprocess 
communication  mechanism  and  later  explore  its  impact  on  structuring  distributed 
programs.  The  University  of  Rochester  Hierarchical  Process  Composition  project  (HPC) 
began  with  a  particular  structuring  mechanism,  and  explored  its  impact  on  interprocess 
communication  [l.eF85b].  [I  eF85a],  Some  of  the  results  are  quite  general,  and  here 
report  an  observation  that  applies  to  most  software  systems  that  emphasize  abstraction  cm 
modularity  and  allow  dynamic  reconfiguration. 

Most  modern  software  systems  build  up  complex  applications  from  a  collection  of 
more-or-less  independent  modules  implementing  different  functions.  In  various  systems 
such  modules  might  be  Ada  packages,  Eden  object,  heavyweight  processes,  etc.,  and  the 
intermodule  (interprocess)  communication  mechanisms  used  to  bind  them  together 
obviously  vary  as  widely.  Although  our  observations  took  place  in  the  specific  context  of 
processes  and  transport  protocols,  we  will  use  the  generic  terms  module  and  1PC 
mechanism. 

There  are  three  distinct  areas  of  responsibility  in  interprocess  or  intermodule 
communication:  signalling,  composition ,  and  implementation.  Th'ese  functions  take 
different  forms  depending  on  the  communication  mechanism  being  used,  but  in  general, 
signalling  defines  the  interactions  between  communicating  peers  (what),  composition 
defines  which  peers  communicate  (with  whom),  and  implementation  provides  the  media 
of  communication  (with  what). 

We  will  not  have  much  to  say  about  signalling,  except  that  it  is  a  completely  distinct 
function  from  composition.  The  interesting  observations  are,  in  a  system  supporting 
abstraction  and  reconfiguration: 

(1)  Composition  is  distinct  from  implementation. 

(2)  Composition  is  an  incremental  and  distributed  function. 

(3)  Implementation  requires  a  third  party  connect. 

2.  Signalling.  Composition,  and  Implementation  are  Distinct  Functions 

It  is  important  to  distinguish  signalling,  composition,  and  implementation.  A 
signalling  operation  is  an  act  of  communication,  like  sending  a  message,  writing  to  a  file, 
or  invoking  an  operation  on  an  object.  A  signalling  sequence  defines  the  basic  behavior 
of  a  process.  Composition  defines  how  basic  behaviors  are  combined  into  a  more 
complex  application,  by  controlling  the  communication  paths  among  its  components. 
Implementation  establishes  the  physical  media  needed  to  support  the  logical  paths 
determined  by  composition. 

While  the  correspondence  is  not  exact,  in  terms  of  the  ISO  seven-layer  model, 
signalling  is  user  invocation  of  the  transport  layer,  composition  is  user  invocation  of  the 
session  layer,  and  implementation  is  carried  out  by  the  session  layer.  Tables  1  and  2 
illustrate  the  signalling  and  composition  operations,  respectively,  for  several  interprocess 
communication  mechanisms.1  These  Tables  are  meant  to  be  suggestive,  rather  than 


Mechanism 

Signalling  Operations 

CONIC 

send,  receive,  reply 

Hydra 

send,  receive,  reply 

RPC 

call,  accept-reply 

socket 

various,  usually  send,  receive 

memory 

read,  write,  P,  V,  fetch-and-phi 

file 

read,  write,  seek 

mailbox 

deposit,  withdraw 

link 

send,  receive 

filter 

send,  receive 

Linda 

insert,  remove,  retrieve,  eval-and-expand 

Table  1. 


Mechanism 

Composition  Operations 

CONIC 

link,  unlink 

Hydra 

connect,  disconnect 

RPC 

bind 

socket 

bind,  listen,  accept,  connect,  disconnect 

memory 

link,  load,  address 

file 

create,  open,  close,  inherit 

mailbox 

create,  name 

link 

create,  transfer 

filter 

set-filter 

Linda 

set-pattern 

Table  2. 

exhaustive. 

As  shown  by  the  Tables,  different  mechanisms  generally  have  different  signalling 
operations.  For  example,  signalling  with  a  message  passing  mechanism  involves  deciding 

1  Descriptions  of  these  mechanisms  may  be  found  in:  CONIC:  (KMS83);  Hydra:  [WLH81]:  RPC: 
[Lis80J.  [NeI81|:  socket:  (CCC70):  mailbox:  [Bri73j,  [BFL761.  [HLG78).  [JCD79];  link:  [BHM77J.  [RaR811. 
[ZwL83j,  (ACF87J  (Accent  terminology  for  link  is  "port",  not  to  be  confused  with  the  Hydra  and  CONIC 
ports);  broadcast  filters:  (FFH73J.  (Ary81).  [KeS84J.  [GKZ85];  Linda:  [Gel85], 
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when  to  send  and  recei\e  messages,  and  what  the  contents  of  messages  should  be. 
Signalling  with  a  semaphore  mechanism  involves  deciding  when  to  wait  ( P )  and  signal 
(O-  Signalling  with  a  remote  procedure  call  mechanism  involves  deciding  when  to  call  a 
procedure,  when  to  return  from  a  call,  and  what  the  arguments  and  return  values  should 
be. 

The  compositional  operations  foi  these  three  example  mechanisms  are  deciding 
whom  to  send  a  message  to  (whom  to  receive  from),  deciding  which  processes  have  access 
to  the  semaphore,  and  deciding  which  client  stubs  are  bound  to  which  server  entries, 
respecti  v  el  > .  The  implementation  of  conventional  interprocess  communication 
mechanisms  is  usually  triggered  directlv  by  composition  and  managed  b>  the  host 
operating  svstem. 

Dividing  an  application  into  manv  modules  reduces  complexitv  b>  providing  manv 
small,  specific  interfaces  with  a  small  number  of  possible  clients.  Interfaces  are  usuallv 
talked  about  as  static  specifications,  but  it  is  a  module's  behavior  or  pattern  of 
interactions  that  is  important,  not  its  entry  points  and  arguments.  Each  module  expresses 
a  behavior  through  its  signalling  with  related  modules,  and  in  systems  that  enforce 
abstraction,  this  behavior  will  not  depend  on  the  identity  of  a  communicating  peer 
module.  That  is,  signalling  should  not  depend  on  composition. 

Dvnamic  installation,  replacement,  or  removal  of  a  module  from  a  running  svstem 
requires  (among  other  things)  the  abilitv  to  change  the  bindings  between  communicating 
modules.  Upgrading  a  svstem  on-line  requires,  for  example,  the  abilitv  to  remove  all 
communication  paths  to  an  outdated  module,  remove  it,  create  a  replacement,  and  then 
create  communication  paths  from  the  old  modules  peers  to  the  new  module.  To  provide 
general  purpose  tools  for  reconfiguring  applications  on  the  fly ,  composition  should  not 
require  the  active  cooperation  of  the  affected  modules.  That  is,  dvnamic  composition 
should  not  depend  on  signalling,  or  on  the  identities  of  the  composed  modules. 

We  are  concerned  mainly  with  how  processes  can  be  dynamically  reconfigured  to 
support  flexible,  long-lived,  surv iv able  applications.  This  means  we  are  not  interested  in 
signalling  per  se.  but  very  interested  in  composition  as  a  logical  operation  and 
implementation  as  a  physical  operation.  The  HPC  project  has  a  specific  notion  of  how  to 
form  logical  communication  paths  between  processes  and  groups  of  processes,  but  our 
observations  will  be  phrased  in  terms  of  more  general  systems. 

3.  Composition  is  Incremental  and  Distributed 

Signalling,  composition  and  implementation  are  not  just  conceptually  different 
activities.  If  distributed  programming  is  to  incorporate  more  complex  abstractions  than 
the  trivial  client-server  model,  these  activities  must  actually  be  carried  out  by  different 
agents.  We  argue  that  composition  must  be  an  activity  distributed  among  multiple 
agents.  Further,  the  composing  agents  are  generally  not  the  same  as  the  signalling  agents. 

We  are  interested  in  software  systems  with  three  significant  properties.  An 
interesting  system  (1)  allows  communication  with  a  group  of  related  modules  (processes) 
as  a  group,  (2)  enforces  the  abstraction  of  a  group,  and  (3)  supports  non-trivial  internal 
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structures  in  a  group.2  In  such  a  system, 

(1)  logical  communication  paths  are  naturallv  divided  into  several  segments,  one  for 
each  group  that  the  path  involves,  and 

(2)  composition  is  a  distributed  activity. 

Readers  who  object  to  "path"  as  too  suggestive  of  physical  routing  are  encouraged  to 
substitute  "binding  decision".  The  difference  between  binding  decision  and  binding  is 
just  that  between  composition  and  implementation. 

There  is  no  shortage  of  interesting  systems.  The  follow  ing  list  of  examples  could  be 
doubled  or  tripled  in  length  easily.  Allowing  for  vagaries  of  notation,  Figure  1  is  a 
natural  illustration  for  the  kind  of  nested  structure  that  can  be  described,  and  in  some 
cases  implemented  directly,  by  each  of  the  example  systems. 

•  distributed  systems  - 

CONIC  [KMS83],  HPC  [LeF85a] 

•  programming  languages  and  environments  - 

DPL-82  [En82],  Piet  [G1T84],  PRONET  [LeM82] 

•  svstem  design,  analysis,  and  modelling  tools  - 

'  DREAM  [Rid8l"],  GRACE/CS  [HaK84],  SADT  [Ros85],  SARA  [FFRS6],  SR  EM 
[AlfS5] 

The  direct  paths  between  modules  at  the  same  level  of  abstraction,  combined  by  the 
paths  inside  modules  that  connect  implementing  children  to  the  external  interfaces  of  the 


J  Either  nesting  of  groups  or  designaung  specific  modules  to  receive  communication  addressed  to  a 
group  sausfies  property  3.  Systems  with  only  single  level  of  process  grouping  and  uniform  internal 
treatment  of  processes  within  a  group  (e.g.,  mulucasting).  such  as  V  [ChZ85],  lack  property  3. 
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module,  naturally  produce  incremental  compositions.  In  most  cases,  different  agents  will 
be  responsible  for  the  design  or  run-time  management  of  different  abstractions,  leading  to 
distribution  of  composition,  as  well. 

Fven  simpler  and  more  common  systems  demonstrate  incremental  composition. 
Consider  a  system  offering  only  one  service  with  multiple  server  processes,  a  file  service, 
say.  Fverv  proces>  either  belongs  to,  or  is  a  client  of,  the  file  service,  and  the 
multiprocess  implementation  of  the  file  service  is  transparent  to  the  clients.  Before  a 
client  and  a  server  process  communicate,  the  client  must  decide  to  access  the  file  service 
and  some  agent  in  the  file  service  must  decide  which  server  process  is  to  handle  the 
client's  request.  Neither  the  client,  nor  the  file  service  agent,  can  decide  unilateral!)  to 
bind  the  specific  client  and  server  processes.  The  logical  path  between  the 
communicating  processes  has  two  segments.  (There  are  two  independent  contributions  to 
the  decision  to  bind  that  particular  pair  of  processes). 

Not  all  our  example  systems  support  reconfiguration,  and  static  structure  avoids 
run-time  composition  altogether.  Composition  is  then  typically  an  activity  of  human 
designers,  while  implementation  is  carried  out  by  configuration  software.  Signalling 
remains  strictly  a  run-time  activity.  However,  static  structure  makes  the  question  of 
distributed  and  incremental  composition  a  moot  issue  of  design  methodology.  VSe 
suggest  that  in  successful  methodologies  composition  remains  distributed  and  increment.!' 
when  the  metric  involves  separation  among  specification  modules  rather  than  process 
groups  or  nested  modules. 

4.  Efficient  Implementation  Requires  a  Third  Party  Connect 

Robust,  flexible,  distributed  software  will  be.  and  is,  designed  as  applications  with 
several  levels  of  internal  abstraction  and  significant,  long-lived,  internal  communication 
patterns.  To  allow  designer  and  maintained  run-time  access  to  the  relationships  among 
an  application's  components,  the  logical  grouping  and  communication  paths  should  be 
explicit  and  persistent  However,  while  this  useful  structure  should  be  kept  at  run-time, 
it  is  not  necessary  or  desirable  to  use  all  of  it  when  manipulating  physical  resources  like 
processes  and  communication  channels. 

CONIC.  HPC.  PRONE!,  and  the  other  examples  operating  at  run-time  provide  a 
value-added  service  by  supporting  intermediate  levels  of  abstraction  and  incremental 
definition  of  communication  paths.  If  all  the  purely  abstract  structure  is  eliminated  by 
erasing  the  abstract  grouping  and  throwing  away  all  the  incomplete  paths,  the  remaining 
structure  consists  of  real  modules  (processes)  and  real  communication  media.  Such  an 
elimination  is  illustrated  in  Figure  2.  Ignoring  abstract  structure  in  this  way  has  no  effect 
on  the  behavior  of  the  system.  No  real  modules,  or  paths  between  real  modules,  have 
been  removed. 

So,  on  the  one  hand,  there  is  the  user  model  with  lots  of  convenient  abstraction,  and 
on  the  other,  there  is  the  physical  environment  with  only  real  pieces.  Some  agency,  for 
convenience  we  will  call  it  an  operating  system,3  translates  the  incrementally  composed 
logical  paths  of  the  user  model  into  the  appropriate  physical  communication  channels. 
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Figure  2. 


This  is  the  third  area  of  responsibility:  implementation. 

Clearly,  the  only  acceptable  implementations  are  those  that  create  physical  media 
directly  between  the  real  communicating  modules,  without  additional  overhead,  and 
regardless  of  the  length  and  complexity  of  the  logical  path  joining  them.  This  is 
practically  a  definition  of  a  third-party  connect.  After  the  operating  system  has  stripped 
away  the  user  lev  el  abstractions,  it  must  be  able  to  connect  and  disconnect  directly  any  of 
the  modules  under  its  aegis.  This  facility  is  critical  for  efficient  implementation  of  all 
multiprocess  software  that  emphasizes  abstraction:  Ability  to  bind  (and  unbind)  modules 
with  a  communication  path  may  rest  with  an  agency  that  never  was,  is,  or  will  be  bound  to 
those  modules  or  associated  with  that  path.  In  other  words,  it  must  be  possible  to  connect 
and  disconnect  two  processes  from  a  third  process. 

5.  Discussion 

5.1.  State  of  the  Art 

It  is  imperative  to  abandon  IPC  mechanisms  where  the  decision  to  connect  processes 
must  be  made  by  the  processes  being  connected.  It  is  unreasonable  to  develop  complex 
multiprocess  software  with  such  mechanisms,  because  they  make  it  impossible  to 
introduce  the  appropriate  use  of  abstraction.  This  is.  sadly,  the  state  of  most  internetwork 
transport  protocols,  like  TCP/IP,  where  communicating  processes  have  to  explicitly 
cooperate  in  establishing  connections.  (To  be  fair,  XNS  Bulk  Data  Transfer  is  an 
internetworking  protocol  designed  from  the  beginning  to  support  third-party  connect 
[Xer84].) 

Another  inadequate  too!  is  inheritance  of  communication  paths  from  a  common 
ancestor.  This  is  the  mechanism  used  by  the  UNIX  shells  to  implement  pipelines,  and  in 
principle,  more  complex  patterns  of  communication  between  processes.  It  does  support 
modularity  and  abstraction,  but  unfortunately  does  not  allow  for  later  reconfiguration  of 
an  application,  even  to  replace  a  failed  process. 


3  This  agency  may  provided  in  any  number  of  different  ways,  depending  on  the  extent  to  which 
applications  can  be  dynamically  reconfigured.  In  the  examples  we  have  cited,  the  agency  is  vanously  the 
operating  system,  a  privileged  server  process,  a  run-time  library,  a  system  generation  utility,  and  a  compiler. 


The  third  common  technique  is  most  clearly  illustrated  by  link  systems,  where  to 
create  a  path  between  processes  A  and  B.  process  C  must  already  have  a  path  to  each  of 
them.  Links  support  modularity  and  reconfiguration,  but  require  the  active  cooperation 
of  the  processes  being  reconfigured  In  particular,  process  C  can  volunteer  a  new  path, 
but  can  not  cause  processes  .4  and  B  to  stop  using  an  old  path.  (A  link  system  allowing 
the  owner  of  a  link  to  transfer  receive  rights  and  to  delete  a  link  even  if  rights  are  held 
by  other  processes  would  not  have  this  deficiency.)  Link  systems  usually  also  use 
inheritance  to  get  over  the  bootstrapping  problem  of  providing  a  process  its  initial  paths. 

None  of  these  techniques,  self-composition,  inheritance,  or  links,  is  an  adequate  tool 
for  the  efficient,  dynamic  reconfiguration  of  modular  software,  yet  there  is  nothing 
fundamentally  difficult  about  providing  third-party  connect,  or  similar  functions.  For 
example,  the  portion  of  the  TCP/IP  protocol  that  is  concerned  with  setting  up  and 
tearing  down  connections  (session  layer  issues)  is  clearly  defined.  It  would  be 
straightforward  to  modify  TCP  to  negotiate  the  connect  and  disconnect  subprotocols  with 
(generally)  a  third  party.  The  server  “passive  mode"  should  be  separated  from  the 
socket/address  multiplexing  facility. 

It  is  no  coincidence  that  modern  system  design  and  analysis  tools  emphasize 
abstraction  and  modularity.  Interesting  systems  in  the  sense  of  Section  3  will  become 
only  more  common.  Clearly,  design  methodology  has  outstripped  the  abilities  of  I  PC 
mechanisms  to  support  it  efficiently.  The  reason  is  perhaps  historical:  software  designers 
are  painfully  aware  of  the  benefits  of  modularity  and  the  pitfalls  of  the  expedient 
solution.  IPC  designers  are  often  trained  in  a  hardware  culture,  specifically  the  long-haul 
circuit  switching  tradition,  with  a  concern  for  performance  and  traditional  services. 
However,  we  can  do  a  great  deal  more  with  IPC  than  emulate  telephones  and  terminals’ 
Designers  of  new  network  protocols  and  IPC  mechanisms  will  be  doing  application 
designers  a  disservice  if  they  do  not  include  session  issues  in  general,  and  third-party 
connect  in  particular,  in  their  considerations. 

5.2.  Authorization 

Given  the  potential  for  unilateral,  uncooperative  change  (of  communication  paths), 
die  question  of  authorization  and  permission  naturally  arises.  To  some  extent,  the  three 
techniques  just  discussed  are  limited  precisely  because  they  want  to  limit  changes  in 
composition  to  implicitly  authorized  processes.  The  HPC  project  provides  a  specific 
answer  to  the  problem  of  authentication  and  authorization,  but  we  do  not  attempt  a 
general  answer! 

Instead,  we  will  note  that  there  are  at  least  two  places  where  the  permissions  for 
created  or  destroying  a  connection  between  processes  can  be  checked.  First,  as 
applications  incrementally  build  up  logical  paths,  their  permission  to  modify  the 
corresponding  pieces  of  (abstract)  structure  can  be  checked.  Second,  as  the  implementing 
agency  attempts  to  create  or  destroy  a  real  communication  channel,  its  permission  to 
modify  that  channel  can  be  checked. 

The  designer  of  internetwork  transport  protocols  will  be  primarily  interested  in  the 
latter  case.  Consider  a  modified  TCP/IP.  When  a  would-be  implementing  agent 


attempts  to  negotiate  a  change  to  a  connection,  the  protocol  software  can  authenticate  the 
agent.  While  the  IP  and  ISO  internets  lack  practical  authentication  services,  the  Xerox 
corporate  internet  has  provided  them  for  several  years.  In  this  way,  the  transport 
protocol  providing  the  third-party  connect  needs  to  know  nothing  about  the  structure  of 
the  applications  in  which  it  is  being  used,  and  the  worker  processes  in  an  application 
simply  have  to  agree  on  a  common  implementing  agent  and  so  inform  the  protocol* 

5.3.  Why  Persistent  Logical  C  onnections  (Compositions)? 

The  primary  objection  to  communication  based  on  explicit,  persistent  connections  is 
traditional,  and  based  on  the  cost  of  establishing  and  destroying  switched  circuit  virtual 
circuits  in  the  telegraph  and  telephone  industries.  To  a  large  extent,  this  limiting 
technology  has  been  left  behind,  even  by  those  industries,  but  stronger  responses  to  the 
claim  of  excessive  cost  can  be  made  when  composition  is  separated  from  implementation. 

First,  changes  in  logical  composition  only  have  an  effect  on  media  when  a  complete 
logical  path  is  created  or  destroyed.  Therefore,  many,  if  not  most,  reconfigurations  will 
affect  only  abstract  structure  and  incur  no  cost  in  set-up  or  tear-down  of  physical  media. 

Second,  many  IPC  mechanisms  are  fundamentally  connectionless.  Consider 
datagrams  where  "connecting"  simply  means  determining  the  destination  address  of  an 
outbound  datagram.  Establishing  a  communication  channel  incurs  no  cost  until  a 
datagram  is  actually  sent,  and  then  no  greater  cost  than  sending  any  other  datagram. 

Thus,  by  separating  composition  from  implementation,  a  software  designer  and 
maintainer  can  have  the  abstraction  of  explicit,  persistent  connections  with  the 
performance  and  cost  of  the  most  appropriate  communication  mechanism(s). 
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